Assessing agreement on classification tasks: the kappa statistic

Currently, computational linguists and cognitive scientists working in the area of discourse and dialogue argue that their subjective judgments are reliable using several different statistics, none of which are easily interpretable or comparable to each other. Meanwhile, researchers in content analysis have already experienced the same difficulties and come up with a solution in the kappa statistic. We discuss what is wrong with reliability measures as they are currently used for discourse and dialogue work in computational linguistics and cognitive science, and argue that we would be better off as a field adopting techniques from content analysis.Comment: 9 page

Modelling Participant Affect in Meetings with Turn-Taking Features

This paper explores the relationship between turn-taking and meeting affect. To investigate this, we model post-meeting ratings of satisfaction, cohesion and leadership from participants of AMI corpus meetings using group and individual turn-taking features. The results indicate that participants gave higher satisfaction and cohesiveness ratings to meetings with greater group turn-taking freedom and individual very short utterance rates, while lower ratings were associated with more silence and speaker overlap. Besides broad applicability to satisfaction ratings, turn-taking freedom was found to be a better predictor than equality of speaking time when considering whether participants felt that everyone they had a chance to contribute. If we include dialogue act information, we see that substantive feedback type turns like assessments are more predictive of meeting affect than information giving acts or backchannels. This work highlights the importance of feedback turns and modelling group level activity in multiparty dialogue for understanding the social aspects of speech

Developing Meeting Support Technologies: From Data to Demonstration (and Beyond)

Proceedings of the 17th Nordic Conference of Computational Linguistics NODALIDA 2009. Editors: Kristiina Jokinen and Eckhard Bick. NEALT Proceedings Series, Vol. 4 (2009), 2-3. © 2009 The editors and contributors. Published by Northern European Association for Language Technology (NEALT) http://omilia.uio.no/nealt . Electronically published at Tartu University Library (Estonia) http://hdl.handle.net/10062/9206

Brief report : imitation of meaningless gestures in individuals with Asperger syndrome and high-functioning autism

Nineteen people with Asperger syndrome (AS)/High-Functioning Autism (HFA) (ages 7-15) were tested on imitation of two types of meaningless gesture: hand postures and finger positions. The individuals with AS/HFA achieved lower scores in the imitation of both hand and finger positions relative to a matched neurotypical group. The between-group difference was primarily accounted for by performance on a test of visual motor integration, together with a hand imitation deficit which was specifically due to errors in body part orientation. Our findings implicate both visuomotor processes (Damasio and Maurer, 1978) and self-other mapping (Rogers and Pennington, 1991) in ASD imitation deficits. Following Goldenberg (1999), we propose that difficulties with body part orientation may underlie problems in meaningless gesture imitation

Inter-Coder Agreement for Computational Linguistics

This article is a survey of methods for measuring agreement among corpus annotators. It exposes the mathematics and underlying assumptions of agreement coefficients, covering Krippendorff's alpha as well as Scott's pi and Cohen's kappa; discusses the use of coefficients in several annotation tasks; and argues that weighted, alpha-like coefficients, traditionally less used than kappa-like measures in computational linguistics, may be more appropriate for many corpus annotation tasks—but that their use makes the interpretation of the value of the coefficient even harder. </jats:p

Planning to Fail, Not Failing to Plan: Risk-taking and Recovery in Task-oriented Dialogue

duplicate the route. The HCRC Dialogue Database [3] We hypothesise that agents who engage in task- contains 128 such dia]0gues; in this work we examined oriented dialogue usually try to complete the task with eight plus a set of dialogues from the pilot study used the least effort which will produce a satisfactory so- in ShadboWs work [17]. Agents who wish to avoid plan lution. Our analysis of a corpus of map navigation task dialogues shows that there are a number of different aspects of dialogue for which agents can choose either to expend extra effort when they produce their initin] utterances, or to take the risk that they will have to recover from a failure in the dialogue. Some of these decisions and the strategies which agents use to recover from failures due to high risk choices are simulated in the JAM system. The human agents of the corpus purposely risk failure because this is generally the most efficient behaviour. Incorporating the same behaviour in the JAM system produces dialogue with more &quot;natural&quot; structure than that of traditional dialogue systems

A simulation of small group discussion

We describe a simulation of turn-taking in small group discussion which includes behaviours from a range of modalities such as speech, gaze, facial expression, gesture, and posture. The simulation is intended to show how these behaviours contribute to the turn-taking process. The agents that participate in the discussion make probabilistic decisions about whether or not to exhibit behaviours such as speaking, making a backchannel, or shifting posture, within a general framework suggested by the existing largely descriptive literature on turntaking. The group behaviours that characterize turn-taking models, such as turns, simultaneous speech, and competition for the oor, emerge from the individual behaviours of the agents. At this stage in the project the basic model has been designed and prototyped

Nonverbal behaviours improving a simulation of small group discussion

This paper reports on the development of a multi-agent simulation of small group discussion that focusses on the interaction and the coordination of turn-taking. We describe the addition of nonverbal behaviours, such as gaze, gestures, posture shifts and head and facial expression, to the model; how the agents in the simulation take these behaviours into account in their decisions to speak, to stop, or to give feedback. The simulation is to be evaluated comparing its statistical profile against the statistics generated by a simpler, base model, one without the nonverbal behaviours, to show that it better approximates the statistics of a real group discussion. The properties to be assessed include mean transition intervals, turn lengths, relation of gaze to speaking order, frequency of simultaneous starts, and of feedback.